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1 Introduction 

All vector spaces, algebras, and modules are over an infinite field F of characteristic charF ^ 2. 
By algebra we always mean an associative algebra. 

The algebra of matrix 0{n) -invariants R'-"-"^ is a subalgebra of the polynomial ring 

Rn = ¥[xij{k) \l<i,j <n,l<k<d], 

generated by (Ti(j4), where 1 < t < n and A ranges over all monomials in matrices Xi, . . . ,Xd, 
X-f, . . . , Xj. Here (Jt{A-) stands for t^^ coefficient of the characteristic polynomial of A and 



X, 



k 



Xii{k) ■■■ Xinik) 



Xnl {k) ■ • • Xnn {k) 



is n X n generic matrix, where 1 < A; < d. Moreover, we can assume that A is primitive, i.e., is not 
equal to the power of a shorter monomial. By the Hilbert-Nagata Theorem on invariants, i?'^'^"^ 
is a finitely generated algebra, but the mentioned generating system is not finite. 

Similarly to matrix 0(n)-invariants we can define matrix GL(n), Sp{n) and S'0(n)-invariants. 
Their generators were found in [16], [13], [1], [4], [18], and [9]. For charF = relations between 
generators for GL{n), 0{n), and ;^(n)-invariants were computed in |15j and [13j . For any charF 
relations for GL{n) and 0(n)-invariants were established in [T7], [TT]. Note that in the case of 
0(n)-invariants we always assume charF ^ 2. 

For f G R denote by deg/ its degree and by mdeg/ its multidegree, i.e., mdeg/ = (ti, . . . ,td), 
where tk is the total degree of the polynomial / in Xij{k), I < i, j < n, and dcg / = ti + ■ ■ ■ + td- 
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Since deg iTt{Yi ■ ■ ■ Yg) — ts, where Yk is a generic or a transpose generic matrix, the algebra 
BP{n) Y^g^ N- grading by degrees and N'^-grading by multidegrees, where N stands for non- negative 
integers. 

Given an N-graded algebra A, denote by the subalgebra generated by homogeneous elements 
of positive degree. A set {0^} C ^ is a minimal (by inclusion) homogeneous system of generators 
(m.h.s.g.) if and only if {oi} are N-homogeneous and {07} is a basis of ^ = A/{A^)'^. If we consider 
a G ^ as an element of then we usually omit the bar and write a ^ A instead of a. An element 
a A \s called decomposable if a = in ^. In other words, a decomposable element is equal to 
a polynomial in elements of strictly lower degree. Therefore the highest degree of indecomposable 
invariants Dmax = -Dmax(fi, d) is equal to the least upper bound for the degrees of elements of a 
m.h.s.g. for R^^^\ So using Dmax we can easily construct a finite system of generators. In this 
paper we give the following estimations on Dmax in case n — i. 

Theorem 1.1. Let n ~ i and d>l. Then 

• // charF = 3, then 2d + 4 < Anax < 2d + 7. 

• //charF 7^ 2, 3, then Dmax = 6. 

A m.h.s.g. for matrix GL(2)-invariants was found [16], [M], and [3] and for matrix 0(2)- 
invariants over F = C in [1^. A m.h.s.g. for matrix GL(3)-invariants was established in [5] and [6]. 
For d = 2 relations for matrix GL(3)-invariants were explicitly described in [T^], [2]. In [TU] Dmax 
was estimated for invariants of quivers of dimension (2, . . . , 2). 

The paper is organized as follows. In Scction[2]we introduce notations that are used throughout 
the paper. We also formulate key Theorem 12.11 from [11^. 

In Section [3] we define an associative algebra A^^d and find out some relations for it. 

In Section [4] we consider relations for R'^^'^^ of multidegree (1, (52 ... , Sd) for an arbitrary n. In 
Lemma ri. II it is shown that the T-ideal of the mentioned relations is generated by two identities, 
whereas in Theorem 12.11 we take infinitely many identities. 

In Section [5] we establish that T-ideal of relations for R'-'^^^ is generated by four identities (see 
Theorem 15.11 together with the definition of ^3,d)- A connection between relations for ^nd 
As^d is described in Corollary [531 

In Section[6]we apply results from Sections [3] and [5] to calculate Dmax and the nilpotency degree 
of A3,d with deviation 3. Note that considering relations from Theorem 15. II of degree less or equal 
than Dmax we obtain a finite generating system for the ideal of relations for R'-'^^K 

2 Notations and known results 

For a vector t— (ti, . . . , t„) G N" we write ff^t = u and |t| = <i + ■ • • + <«. In this paper we use the 
following notions from [llj : 

• the monoid M (without unity) freely generated by letters Xi, . . . , Xd, xj" , . . . , xj, the vector 
space A4f with the basis A4, and M C Ai the subset of primitive elements, where the notion 
of a primitive element is defined as above; 

• the involution : A4f A4f defined by x'^'^ — x for a letter x and (oi • • • Op)"^ = aj • • • 
for ai, . . . , flp e A^; 
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• the equivalence yi ■ ■ - yp zi ■ ■ ■ Zp that holds if there exists a cyclic permutation n £ Sp 
such that 2/^(1) • • • = zi---Zp or • • • = zj • • • zf , where yi, . . . , yp, zi,...,Zp 
are letters; 

• a ring with unity of (commutative) polynomials over F freely generated by "symbolic" 
elements (Tt(a), where t > and a G A4w; 

• 7V(T, a ring with unity of (commutative) polynomials over F freely generated by "symbolic" 
elements (Tt(a), where t > and a £ Af ranges over ^-equivalence classes; note that A/'o- — 
Aia/L, where the ideal L is described in Lemma 3.1 of 

• N-gradings by degrees and N''-gradings by multidegrees for M, Mw, A/", and A/'^, where 
mdeg(a) = (deg^^ a + deg^r a,..., deg^^ a + deg^j a) ior a e M; 

• the surjective homomorphism '■ ■N'a i?'^*-"-', defined in the natural way (see Section 1 
of HI]); 

• tTt,r(a, 6, c) G A/'ct introduced in [12], where a, 6, c G TWf and t, ?- e N (see Section 3 of [IT] or 
see Definition 12.31 for the image of at.r{a, b, c) in Ma); 

• a partial linearization (Jt-r:siQi',b; c) G Ma of at^rio.,b, c) (see Section 5 of [H]), where t = 
(ti,...,t„) G N", r = "(rr,...,r-.) G N^ s = (si,...,s^) G N"' satisfy |r| = a = 
(ai, . . . , a„), 6 = (6i, . . . , &t;), c (ci, . . . , c^), and a^, bj,Ck belong to A^f for 1 < z < u, 
^'^j^'v,l<k<w. A formula for computation of <yt-r-siQ.',b;c) is given in Lemma 5.2 
offTl]. 

The mapping induces the isomorphism of algebras 

RO(n) ^ JTa/^ 

for some ideal if„ <A/'o-. Elements of Kn are called relations for The ideal of relations was 

described in Corollary 7.6 of [TT] : 

Theorem 2.1. The ideal of relations Kn is generated by ot^rifl, b, c), where t + 2r > n and a, b, c 
range over Mp. 

Remark 2.2. Since F is infinite, we can reformulate Theorem 12.11 as follows: the ideal Kn is 
generated by N'^-homogeneous elements o't-r-sifEk; c), where \t\ + 2\r\ > n and a;, bj, Ck range over 
M. 

Consider a = yi • • • j/p, where yi, . . . , j/s are letters. We say that a letter yi is followed by yj 
in a if j = i + 1 or j — 1, i — p. For the sake of completeness let us recall the definition of 
at,r{a,b,c) gJJ^. 

Definition 2.3 {of (Jt,r{a, b, c) G Ma)- We assume d > 3. Denote by It,r — {ak} the set of primitive 
pairwise different words in letters Xi,xf , i = 1,2,3, satisfying 

• jak mdeg(afc) = {t, r, r) for some ja^; 
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• the letters xi,X3, x'^ are followed by xi,X2,X2 in ak; 

• the letters xf,X2,X2 are followed by xf,X3,x'^ in ak- 
Then 

crt,r{xi,X2,X3) = ^ (-1)*° crj^(a) in A7^, 

where ^.a — t + ja{deg^^ a + deg^^ a + 1). We also set crofi{xi,X2,X3) = 1. For any d > 1 and 
01,02,03 e Mw we define (74,^(01,02,03) G A/^ as the result of substitution 
i = 1, 2, 3, in at,r(xx,X2,x-i) £ JJ^. 

The decomposition formula from ^TJ implies that for n x n matrices Ai, i = 1,2,3, with n — 
to + 2r, to > we have 

to 

DP,,,(Ai + A£;,A2,^3) = ^A*«-Vt,.(Ai,A2,A3), (1) 

t=0 

where DPr,r(^i, ^2, ^3) stands for the determinant-pfaffian (see [8J) and crt.,.(Ai, A2, A3) is defined 
as the result of substitution — > Ai, of — > Aj in cri_r(oi, 02, 03). Thus DPr.r relates to at^r in 
the same way as the determinant relates to at- 



li f,h £ Ma- are equal as elements of i?*-**^"), then we write f = h. In particular. / G Ma- is a 
relation for 7?*^^") if and only if / = 0. We say that f = h follows from /i = hi, . . . , fs = hg, if 
/ — /i is a linear combination of fi — hi, . . . , fs — hg in A/^. We use the following convention: 

tr(a) = cri(a) 

for all o S A4w. Note that tr is linear in Afa, i.e., tr(aa + (3b) = a tr(a) + (3 tr(6) in Afa for a, /? G F 
and a, 6 G A^f (see Lemma 3.1 of [H]). Let us remark that 

<^t-r:s{3i'^t^&) = '^t;s;ria^;c;b) in A/'ct, (2) 

where gT' — {of , . . . , aJ). 

3 The algebra A^^^i 

Given a G Aiw, we denote a = a — . For an algebra A, denote by id{ai, . . . ,Os} the ideal 
generated by ai , . . . , £ A. We denote 

• N3^d = M¥/id{a^ \ a £ Mw}; 

• T{a, b, c) = abc + ba^c + bca = abc + bac + bca — bac, where a,b,c £ Air; 

• As.d = N3,d/id{T{a, b, c)\a,b,c£ A4f}; 

• AAi — AiU {1}, i.e., we endow Ai with the unity. 
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Since F is infinite, the elements 

• Ti{a)=a^ 

• T2{a, b) = a^h + aba + ba^, 

• T^{a, b, c) — abc + acb + bac + bca + cab + cba, 
where a, 6, c G A^f, are equal to zero in N^^d- Moreover, 

N3,d - A^F/id{Ti(a), T2(a, 6), T^ia, b, c)\a,b,ce M}. 
In what follows we apply the following remark without references to it. 

Remark 3.1. Consider / = aiUibci, where G F, a^, Ci G Mi, and b G M. If / = in A^^d 
is valid for all b G Ai, then / = in A^ d is also valid for all b G A^f- 

The following equalities in M are trivial: 

a = 2a, = —a = , ab — ab — ba + ba + ba — ba. 

Lemma 3.2. We have 

a) If a G A4 and x is a letter, then a G A^ d is equal to the sum of the following elements: 
bi, 61x62, 61x^62, bix^cxb2, where 61,62 G -M-i, deg^{bi) = deg^ribi) — for i — 1,2, and 
c G M; in particular, i/degj. a > 3, then a — in A^^d,' 

b) //charF — or charF > 3, then xi ■ ■ ■ xq ~ in A^^d,' 

c) {abY = b^a^ in A^^d, where a,b ^ Ad; moreover, (ai ■ ■ ■ asY = ' ' ' ct? ^3,d, where 
ai, . . . , fls G M; 

d) ab ■ c ■ ba — —ca^b^ — b^a^c in A^ d, where a,b,c ^ M; 

e) //charF = 3 and for a ^ A4 we have deg^j^(a) = deg^^{a) = 3, then a — J2i o^i aix\x'^xiX2bi 
in A'i^d, where G F and Oi, bi G A^i; 

f) abc = —cba in A3 d, where a,b,c £ AA. 

Proof, a) Let cE Ad. Considering T2{x,c) we obtain 

xcx = —x^c — cx^ in ^3^^, (3) 
and considering T2{x,xc) we obtain 

xcx^ = —x^cx in AzA- (4) 

The required follows from ([3]) and ([4]). 
b) See Proposition 1 of [S]. 
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c) The first part follows from part a) of Lemma 13.21 and the second part is a consequence of 
the first part. 

d) Applying part a) of Lemma we obtain ab-c-ba = a'^b^c + cb^a'^ + a^cb"^ + b'^ca?' in A^^d- 
To complete the proof, we consider T^{a? , b^,c). 

e) See Statement 7 of [S]. 

f ) The equality T{b, a, c) + T{b, c, a) — T^ib, a,c) — Q in ^ gives the required. □ 

Lemma 3.3. Let charF = 3 and 1 < i < d. For any homogeneous e S Mw of multidegree 
(^1, . . . , 5d) with (5i < 3 and (5i + ■ • • + (5,;_i + 5i-\-i + • • • + (5^ > we define 7ri(e) G o,s the result 
of substitution Xi —> 1, xj — > 1 in a, where 1 stands for the unity of J\Ai. 
Then e = in A^^d implies TTi{e) = in A^^d- 

Proof. Let a,b,cG A4w- By definition, 'Ki{ab) — 'Ki{a)'Ki{b). It is not difficult to see that 7ri(a) — 
■Kiia). Then by straightforward calculations we can show that 'Ki{T2{a,b)) — 0, ■ni{Tj,{a^b,c)) — 0, 
and 'Ki{T{a, b, c)) = in A^^d- The proof is completed. □ 

Lemma 3.4. The equality 

aubvcwe = 

holds in A^ d for all a,b, c, e € A4 and u,v,w € A4i. 

Proof. In this proof all elements belong to Ai and all equalities are considered in A^^d- 
Consider T{ab, c, e) = 0. Using ab = ab — ba and part f) of Lemma 13.21 we obtain 

abce = 0. (5) 

Equalities T{u, a, b)ce — and abT{u, c,e) — together with ([5]) imply 

aubce = —abuce = abcue. (6) 

Hence aT{u, 6, c)e = implies 

aubce = abuce = abcue = 0. (7) 
It follows from T-^iau^bc^ve) — together with ([5]) and ^ that 

aubcve = 0. (8) 
Thus auT^ib, v, ce) ~ and T^iab, u, c)ve = imply 

aubvce = abucve — 0. (9) 
Considering T^{au,bvc^we) — 0, we obtain 

aubvcwe = 0. (10) 

The claim of the lemma is proved. □ 

Lemma 3.5. Let charF = 3. For < s < 4 we assume that xi, . . . ,X8-2s are pairwise different 
letters, ai, . . . , as,v,w E A4, and ui, . . . , Us+i £ Mi. If deg^,. (ui • • • u^+i) ~ 3 for 1 < i < 8 — 2s, 
then the following equalities hold in A^^d-' 
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a) MiOi • • • UsGsUg^i = 0; 

b) a product of vw — wv, Oi, . . . , a^-i, ui, . . . , Ug+i in any succession is equal to zero, where 
s > 0. 

In particular, i/ u G Aii and dcg^. (u) — 3 for 1 < i < 8, then u — 0. 

Proof. In this proof all equalities are considered in A^ ^. We prove by decreasing induction on s. 

1. Let s — 4. Then part a) follows from Lemma \TM 

2. Assume that part a) holds for some s > 0. We set Os — vw in part a). Thus, 

MlOi • • • Us-ia,s-lUs{vW — wv)us+i — 0. 

Considering with 1 < z < s instead of Os, we complete the proof of part b) for the given s. 

3. Assume that part b) holds for some s = fc + 1 > 0. We claim that part a) is valid for s = k. 
We set W = x\x\xiX2- By part e) of Lemma f3.2[ uiOi ■ ■ -UkCLkUk+i is a linear combination of 
elements 

ViWi- ■ ■Vk+lWk+lVk+2, (11) 

where pairwise different Wi, . . . , Wk+i belong to the set {ai, . . . ,ak, W} and Vi, . . . , Vk+2 range 
over M-i. For v = x\x'^ and w = Xi v^e have W = vwx2 and wvx2 = 0. Part b) for s — k + 1 
implies that all elements from (jlip are zero and the claim is proved. □ 



4 Relations for i?*^*^") of multidegree (1, ^2 • • , ^d) 

In this section we assume that n is arbitrary. 

Lemma 4.1. Any homogeneous relation of i?*^'") of multidegree (1, 52 ... , Sd) follows from rela- 
tions: 

• o-i.t-i;r;r(e, a;6;c) = 0, 

• crt;i,r--i;r(a; e, 5; c) = 0, 

where t + 2r = n + 1, a,b,c range over Aif, and e ranges over elements of M such that deg^.^^ (e) + 
deg^f (e) = 1- 

Proof. We assume that / — (Jt-.r-sis^'^^'iS) is a homogeneous relation of multidegree (1,(^2 . . • ,<5d), 
where |t| + 2|r| > n, ai,bj,Ck G M, and consider the set of all ti, rj,Sk. Then at least one element 
of this set is equal to 1. We set u — #a, v = #6, and w = #c. 

If = 1, then applying Lemma 5.2 of [TT] it is not difficult to see that the next equality holds 
in JJ^: 

U V 

f = - ^crA.(aiai, 02, ... ,a„;&;c) - ^ cre^ (02, . . . , a^; oi&j , 61, . . . c)+ 
i=2 i=i 

V 

+ ^o-e^ (02, ■■■ ,au; aibj, 61, ... , 6«;c), 
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where = {l,t2, ■ ■ ■ ,ti ~ 1, . . ■,tu;r;s) and 9^ ^ {t2, ■ ■ ■ ,tu;l,ri, . . . ,rj - 1, . . . ,ry; s). 
If ri = 1 , then the next equahty holds in Ma- 

u w 

f = -^CTA, (a;6iaf ,62, . . . ,6«;c) - ^0-0^(610^,01, . . . , a„; 62, . . . ,6„;c)+ 

i=l fc=l 

+ X! (^I'^fe ' ^1, • ■ ■ , &2, ■ • • , c), 

fc=i 

where Ai = {ti, . . . ,ti - 1, . . . ,tu;l,r2, ■ . . ,r^; s), Qk = (1, ti, . . . , t„; r2, . . . , r^; si, . . . , Sfc - 

1, . . . , Siij ). 

All other cases can be treated similar. We repeat this procedure and use ([2]) to obtain the 
required. □ 



5 Reduction to 743 ^; 

In this section we assume n = 3. This section is dedicated to the proof of the following theorem 
and its corollary. We say that / G Aif does not contain a letter x if / is a linear combination of 
ai, . . . , fls G A^, where deg^. at — for all i. 



Theorem 5.1. Any relation for R'-'^^^ follows from relations: 

(A) tr(au) = for u G A^f such that u — in A^^^; 

(B) (73(a5) = 0; 

(C) at{a) = Q forty 3; 
where a,b ^ Ai. 

Lemma 5.2. 

a) (T2(a) = — itr(a^) and Sa^ia) = tr(a"^) in Ma, where a G A^f/ 

b) A relation h = 0, where h has multidegree (1,(52, • ■ ■ ,<5rf), follows from (A). 

c) A relation at^rio-, b, c) = 0, where i + 2r > 3, a, 5, c G A^F; and t or r is not divisible by charF 
(or charF — 0), follows from (A). 

d) If a (z A4 and deg^.^ a > 3, then tr(a) = follows from (A); 

e) //charF = 3 and u ^ in A^ d, then tr(u) = follows from (A); 

Proof, a) Since tr(a^) = tr(a)^ — 2a2{a) in TVo- (see Lemma 3.1 from [H]), we obtain the first 
equality. The second equality is can be proved similarly. 

b) By Lemma 14. 1[ it is enough to show that /i = ai^t-i-r-r{s, a;b] c) = and f2 = 
o't;i,r-i;r{a',e,b;c) = with t + 2r = 4 follow from (A), where a,b,c G A^f and e G Ai. To 
complete the proof we consider the following cases. 
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If t = 2 and r = 1, then /i = — tr(eT(a, b, c)) and /2 = — tr(eT2(a, c)) + tr(er(a, a, c)) in A/^. 

If < = 4 and r = 0, then /i = - tr(ea^) in A^. 

If t and r = 2, then /2 = - tr(ec6c) = - tr(er(c, 6, c)) in JJ^. 

c) If i is not divisible by charF, then df b, c) — j(Jt-i.i;r;r{a., a; b; c) (see Lemma 5.5 of TT). 
Part b) concludes the proof. The other case is alike. 

d) It follows from the proof of part a) of Lemma [3?2l and linearity of tr. 

e) For a,b,c e M we have tr(r2(a,6)) = 3tr(a^6), tr(r3(a, 6, c)) = 3tr(a6c) + 3tr(ac5), and 
tr(T(a, b,c)) — 3tr(a6c) in A/'o-. By part a) of Lemma [5T^ we have tr(ri(a)) = 3(73 (a) in 77^. The 
proof is completed. □ 



Lemma 5.3. We have that (A) and (B) are relations for R'-"^'^y 

Proof. The proof of part b) of Lemma \S7]\ implies that tr(ea'^) = and tr(eT(o, 6, c)) = in R'-"^'^^ 
for all a,b,c ^ Air and e € A4. The fact that tr is linear shows that (A) is a relation. 

Since a3{AB) = a3{A)a3{B) for all 3 x 3 matrices A and B over the polynomial ring R (see 
Section [1]), we obtain 0-3(06) = 0. □ 



Lemma 5.4. Relations cro,3(w, w) = and a^^^iu, v, w) = 0, where u,v,w 6 A4w, follow from (A), 
(B), and (C). 

Proof. Clearly, without loss of generality we can assume that u = xi, v = X2, w = X3 are letters 
and d > 3. Consider t,r > such that t + 2r > 3 and i or r is odd. 

Let a; e A4 be a letter and a = fcoii^i ■ ■ ■ o,pbp G A4, where ai, . . . , S M are words in 
x,x^ and 61, . . . ,6p-i € A4, bo, bp S A^i do not contain x,x'^, i.e., deg^(6i) — deg^r(6i) — for 
< i < p. We say that a is fixed by x if there is no non-trivial cyclic permutation n € Sp such that 

T T 

oi — a7r(i) , . . . , flp — a-rr^p) or — a„(^p), . . . , — a7r(i) ■ 

Assume that a = boaibi ■ ■ ■ Opbp is fixed by x and bo ■ ■ - bp = yi ■ ■ ■ ys, where yt is a letter for all i. 
Then the following elements 

coaici • • ■ flpCp , 

where ci, . . . , Cp_i S M, cq, Cp G A/Ji, and cq • ■ • Cp = Zp(i) • • ■ Zpf^^-j for £ {j/j, } (1 < i < s) and 
p G Ss, are primitive and pairwise different with respect to the ^-equivalence. 

For x G {a;i,X2,a;3} we denote Y = {x2,X3} \ {x}. The above mentioned property together 
with the definition of at^r implies that 

CTt,r(a;i,a;2,X3) = /i +/2 +/3 in A/^. (12) 

Here 

• fi — tr(a)|j,^y. yey, where the sum ranges over a e It^r such that a is fixed by x 
and degj^(a) ~ r for all ?/ G F; 

• f2 — tr(a), where the sum ranges over a G It,r such that a is not fixed by x; 
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• /a = follows from (B) and (C). 

Let us consider ut^r from the formulation of the lemma. 

a) Let i = and r — i. We take x = X2- By part c) of Lemma [3?2l 

/i = tr(x2a:3a:2a;3a;2 xa) = tr(x3a;2a:2 ^3) = ^-^id 

/2 = ir{x2X'iX2X3X2x'^) - tT{x2X3X2xJ X2xJ) = tr(a;3a;2a;2a;^f ) - tr(a;2a;3(a;3 )^a;2) = 
follow from (A). 

b) Let t ~ r = 3. We take x — xi. By Lemma [S^U fi=0 follows from (A). Clearly, a G It.r 
is not fixed by x if and only if a ~ xaixa2xa3, where ai, 02, 03 £ A4. Thus, f2 = hi + /i2 in JVa, 
where 

hi = iT:{xix'^ x'^ xix'^ Xj,XiX2xT^) + ti:{xix'^ x'^ XiX2x'^ xix'^ x^) 
— tT:{xix'^x'^Xix'^X3XiX2Xz) — ir{xix'^x'^XiX2x'^x 1X2X3) 

~ tT:{xix'^ x'^ XiX2X3Xix'^ X3) ~ ix{xix'^x'^XiX2X3XiX2x'^) 
+ ix{xix'^X-iXiX2x'^XiX2X3) + ir{xix'^X-iXiX2X-iXiX2x'^) 

and /12 is a linear combination of elements h — tr{xiy2y3Xiy2y3XiZ2Z3), where yi, Zi G {xi, xj} for 
i = 2,3. Since Xiy2y3 ■ Xiy2y3 ■ xi = • xi = in ^3 ,^ (see part c) of Lemma [3.2p . h = 

follows from (A). 

Using the equality b'^ = b — b, we can get rid of and x'^ in hi. Hence 

hi = tr(a;iX2X3XiX2a;3a;iX2X3) + tr(a;iX2X3a;iX2X3a;iX2a;3) 
+ tr(a;iX2a;3a;ia;2X3a;iX2a;3) + tr(a;iX2a;3a;iX2a;3XiX2X3) + /13 

in A/'cr, where ^13 is a linear combination of elements h as above. By Lemma l3.4( 

tr(a;iX2X3a;iX2a;3a;iX2X3) + tr(xiX2i3a;ia;2X3XiX2a;3) = 

follows from (A). Applying part d) of Lemma [3.21 to X3 ■ X1X2 ■ X3 ■ X1X2 ■ X3 and using part c) of 
Lemma |3.2[ we obtain 

tr(xia;2a;3XiX2a;3Xia;2a;3) + tr(a;ia;2a;3Xia;2a;3a;i2;2a;3) = — tr(a;ia;2a;3a;3a;2a;^) = 

follows from (A). The proof is completed. □ 

Proof of Theorem \5.1[ By Theorem 12 . 1 1 and Lemma [5731 it is enough to show that / ~ a-t.ria, b, c) = 
0, where t + 2?- > 3 and a,b,c ^ Mw, follows from (A), (B), and (C). 

Applying part a) of Lemma l5.2[ we obtain that / = /i + /2 + /3 in JVa, where fi = ai tr(ai), 
/2 — J2j (^j'^si^j)^ cti^Pj G a'i^bj e M, and /3 follows from (C). Since deg/ > 3, /2 = follows 
from (B). 

If t > 6 or r > 6, then /i = follows from (A). To prove this claim, note that if t > 6, 
then degjj > 3 or deg^^r ai > 3, where we consider ai as a word in a, b, c, a'^ , b^ , . Part d) of 
Lemma [nH] implies that tr(ai) = follows from (A). 

a) Let charF = or charF > 5. Part b) of Lemma 13.21 implies that if deg/ > 6, then 
/i = follows from (A). If deg/ < 6, then / = follows from (A) by part c) of Lemma [5.21 or 
f = cr^{x) =0 follows from (C), where a; is a letter. 
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b) Let charF = 3. In the rest of the proof we apply part e) of Lemma without reference to 
it. We have /i = J2k Tfe tr(cfc), where 7fc G F and Ck is a monomial in a, b, c, a, b, c for all k. 

We assume r = 6. If degf, Cfc = deg^Cfc = deg^ Cfc = deggCfc = 3, then tr(cfc) = follows from 
(A) by Lemma otherwise, tr(cfc) = follows from (A) by part d) of Lemma [3T^ 

We claim that if t = 6 and r = 3, then tr(c/c) = follows from (A) for all k. Assume that 
this claim does not hold. Then degjj(cfc) = degg(cfc) = 3 by part d) of Lemma and degb(cfc) = 
degc(cfe) = 3 by Lemma [3^ Part e) of Lemma [3T2l implies that Ck = aWaoWhc + PWbcWaa m 
A^^d, where a,/3 G F and Wuv = u^v'^uv for any u,v ^ M^. We set v = 6^c^, w = b, and s = 4 in 
part b) of Lemma [531 and obtain WaoWhc = WbcWaa = in A^^^d- Hence tr(cfe) = follows from 
(A); a contradiction. Thus the claim is proved. 

lit — Q and r = 0, then it is not difficult to see that / = (yQ{a) = follows from (A), (B), and 
(C) by Lemma 9 of 5 . 

Part c) of Lemma [5^ and Lemma WM complete the consideration of the case of charF = 3. □ 

Corollary 5.5. Let u,v G Air do not contain xi,xf . We have 

a) tr(uxi) = i/ and only if u ~ Q in ^3^^; 

b) i/ charF = 3, then tr(Ma;^) = if and only if u = Q in A^^d- 

Proof, a) If u = in A^^d, then tr(uxi) = by Lemma 15.31 

If tr(ua;i) = 0, then tr(MXi) = '^■aitr{uiai) in 77^, where ai £ ¥, Oi £ A^, Ui £ Alp is 
N'^-homogeneous with = in ^3,^, and deg^^{aiUi) + deg^j (aiUi) — 1 (see Theorem 15. II) . For 
a, b,c,e£ Aiw the following equalities in Afa- hold: 

tr(r2(a, b) e) = tr(T2(a, e) b), 

tr(T3(a, b, c) e) = tr(T3(e, b, c) a), 
tr(T(a, 5, c) e) = tr(T(e, 6, c) a), 
tr(T(a, 5, c) e) = tr ((r3(a, e, c) — T(a, e, c) — T(e, a, c)) b) , 
tr(r(a, 6, c) e) — tr (^{T^{a, b, e) — T{a, b, e) — T(e, b, a)) c) . 

Thus tr(ua;i) = J^j Pj ir{vjXi) in J\fa, where /3j £ F and Vj £ Mr satisfies vj = in A^ d- Therefore 
u — PjVj in A4w and the proof is completed. 

b) If ti(uxi) = 0, then tr(M (xi + Xd+i)'^) = 0, where Xd+i stands for a new letter. Taking 
homogeneous components of multidegrees (1, (52, ... , Sd, 1) we obtain tr((aa;i + xia)xd+i) = 0. By 
part a), uxi + xiu = in A^^d- Application of tti from Lemma 13.31 completes the proof. □ 
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6 Dmax and the nilpotency degree of ^43 ,^ 

Theorem 11.11 is a consequence of Lemma 16.31 which is proved in this section. For an n x n matrix 
X we denote X = X - X^ . 

Lemma 6.1. The invariant tr(X^X^XiXi) £ R'^^^^ is indecomposable, i.e., tr(a;f i^xiXi) ^ 0. 

Proof. For short, we write X instead of Xi. Assume that the claim of this lemma does not hold. 
By part d) of Lemma [5T21 any indecomposable element of R'-'^^^ of multidegree (fc), where fc < 6, 
is a linear combination of the following elements: 

tr{X), a2iX), aaiX), tr(XX^), triX^X^), tT{X^{X^)^), a2{,XX^). (13) 

Then 

tr(X2x2xX) = ^ (14) 

■i 

where G F and fj is an element from Moreover, if we take an arbitrary 3x3 matrix over R 
instead of X, then remains valid and coefficients do not depend on X . Denote by a, /?, 7 S F, 
respectively, the coefficients of tr(XX^)3, iY[XX^)tY{X^{X^f ), and iY{XX^)(T2{XX'^), respec- 
tively, in p4)) . For a,b £ R we set 

/ a \ 
X = I b \ . 

V / 

If we take 6 = 0, then a = 0. If we assume a, 6 ^ 0, then 

atr{XX^)^ + f3tr{XX'^)tr{X'^{X'^f) +jtr{XX^)(72iXX'^) = 0. 

Thus 

a%\l + /3 + 7) + a^b\p + 7) = 
for all a,b £ R; sl contradiction. □ 

Lemma 6.2. Let charF = 3. Then 

a) for any a £ A4 satisfying a ^ in A3 we have dega <2d+7. 

b) there exists an a £ A4 such that a^Q in A^^d o,nd dega = 2d + A. 

Proof. a) In this proof all equalities are considered in A^^d- We can assume that a is a word 
in xi,. .. ,Xd, xi,. .. ,Xd. By part a) of Lemma [3.21 deg^,. (a) < 3 for all i. Let k = deg^^ (a) + 
• • ■ + degj^(a). By Lemma [3.41 k < A. Moreover, by part a) of Lemma [3.51 there are no pairwise 
different 1 < Ji, . . . ,i8-2k < d such that deg^.. (a) — 3 for all 1 < < 8 — 2fc. Thus, dega < 
k + 2d+ {7 -2k) <2d + 7. 

b) Consider Od — x\x\xiXiX2 ■ ■ ■ x"^ £ M. If = 1, then ad 7^ in A^^d by part e) of Lemma l5.2l 
and Lemma [6. 11 We assume that d > 1. If a^ = in ^3,^, then applying Tr^, . . . , 7r2 from Lemma [3.31 
to ad we obtain that ai = in A^ d] a contradiction. □ 

Lemma 6.3. 
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a) //charF = 3 andd>l, then 2d + A < D^ax < 2d + 7 . 

b) //charF ^ 2 and d ~ 1, then D^^s.^ = 6. 

a) //charF 7^ 2,3 and d > 1, then D^i^^ — 6. 

Proof, a) Let tr(a) ^ 0, where a G AIf- By part e) of Lemma [5.21 a ^ in A^^d. Part a) of 
Lemma \6l2\ imphes dega < 2d+7. Thus part a) of Lemma [5?2l and cquaUty (B) from Theorem l5.1l 
show that Dmax <2d + 7. 

Consider ad — x\x\xixix\ • • ■ x\ G M. If d = 1, then tr(ad) ^ by Lemma lOl We assume 
that d > 1. If ir{ad) — 0, then ad-i = in A^^d by part b) of Corollary 15. 5| a contradiction with 
the proof of part b) of Lemma W% 

b) Let a e and d = \. If dega > 6, then deg^^(a) > 3 or deg^T(a) > 3; and part a) of 
Lemma 13.21 implies a = in A^ d- Lemma |6. II completes the proof. 

c) The required follows from part b) of Lemma l3.21 equality (A) of Lemma lS-H and Lemma l6.1l 

□ 



Remark 6.4. Denote by -Dnii the nilpotency degree of ^3^^, i-e., the minimal s such that ai • • • = 
in A^ d for all ai, . . . , G A4f. It is not difhcult to see that if ci > 1, then the same estimations 
as for -Dmax ill Lemma 16.31 are also valid for D^^n — 1 . 

Hypothesis 6.5. If charF = 3 and d> 7, then -D„iax — 2d + 7. 

Remark 6.6. Let charF = 3 and d>7. To prove Hypothesis 16. 51 it is enough to show that 

xlW23XiWi5We7 ^ in As^d, (15) 
where Wij This claim follows from part b) of Corollary 15 . 51 and Statement 8 of [5]. 
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